Boosting with the L 2 -loss: Regression and Classiication

This paper investigates a computationally simple variant of boosting, L 2 Boost, which is constructed from a functional gradient descent algorithm with the L 2-loss function. As other boosting algorithms, L 2 Boost uses many times in an iterative fashion a pre-chosen tting method, called the learner. Based on the explicit expression of reetting of residuals of L 2 Boost, the case with (symmetric) linear learners is studied in detail in both regression and classiication. In particular, with the boosting iteration m working as the smoothing or regularization parameter, a new exponential bias-variance trade oo is found with the variance (complexity) term increasing very slowly as m tends to innnity. When the learner is a smoothing spline, an optimal rate of convergence result holds for both regression and classiication and the boosted smoothing spline even adapts to higher order, unknown smoothness. Moreover, a simple expansion of a (smoothed) 0-1 loss function is derived to reveal the importance of the decision boundary, bias reduction, and impossibility of an additive bias-variance decomposition in classiication. Finally, simulation and real data set results are obtained to demonstrate the attractiveness of L 2 Boost. In particular, we demonstrate that L 2 Boosting with a novel component-wise cubic smoothing spline is both practical and eeective in the presence of high-dimensional predictors.